Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](inverted index) inverted Index File Cache Queue Optimization #46024

Merged
merged 1 commit into from
Jan 3, 2025
Merged

[fix](inverted index) inverted Index File Cache Queue Optimization #46024

merged 1 commit into from
Jan 3, 2025

Conversation

zzzxl1993
Copy link
Contributor

@zzzxl1993 zzzxl1993 commented Dec 26, 2024 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

  1. All files involved in the process of opening the reader for the inverted index are added to the index queue of the file cache.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zzzxl1993
Copy link
Contributor Author

run buildall

@zzzxl1993
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32690 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4cb8fc8d501d7864772d6f9095f85b7a41c429b3, data reload: false

------ Round 1 ----------------------------------
q1	17574	6347	6062	6062
q2	2050	313	181	181
q3	10417	1231	774	774
q4	10257	859	433	433
q5	8485	2200	2021	2021
q6	208	182	152	152
q7	900	747	602	602
q8	9264	1410	1181	1181
q9	5324	4861	4913	4861
q10	6771	2316	1867	1867
q11	485	281	257	257
q12	350	389	227	227
q13	17787	3556	2911	2911
q14	246	241	220	220
q15	564	502	482	482
q16	638	611	599	599
q17	592	859	322	322
q18	7036	6553	6426	6426
q19	3034	959	554	554
q20	303	320	185	185
q21	3037	2265	2063	2063
q22	379	338	310	310
Total cold run time: 105701 ms
Total hot run time: 32690 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6378	6202	6206	6202
q2	237	338	229	229
q3	2252	2665	2367	2367
q4	1422	1825	1358	1358
q5	4358	4775	4973	4775
q6	184	177	151	151
q7	2119	1942	1834	1834
q8	2627	2828	2781	2781
q9	7314	7299	7241	7241
q10	3116	3386	2731	2731
q11	577	508	500	500
q12	705	796	623	623
q13	3349	3798	3108	3108
q14	291	308	296	296
q15	561	523	507	507
q16	646	686	669	669
q17	1238	1742	1250	1250
q18	7644	7546	7510	7510
q19	853	1216	1106	1106
q20	2037	2014	1950	1950
q21	5782	5203	4975	4975
q22	658	614	611	611
Total cold run time: 54348 ms
Total hot run time: 52774 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197685 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4cb8fc8d501d7864772d6f9095f85b7a41c429b3, data reload: false

query1	1294	940	911	911
query2	6492	2376	2309	2309
query3	10959	4778	4785	4778
query4	32892	23695	23923	23695
query5	4302	612	455	455
query6	278	194	181	181
query7	3983	492	303	303
query8	283	231	225	225
query9	9348	2696	2690	2690
query10	463	315	248	248
query11	18158	15321	15080	15080
query12	170	113	104	104
query13	1586	559	425	425
query14	9395	7494	7338	7338
query15	254	204	194	194
query16	8187	627	466	466
query17	1561	765	591	591
query18	2163	423	326	326
query19	201	203	176	176
query20	131	131	121	121
query21	206	129	113	113
query22	4650	4619	4552	4552
query23	35141	34023	33809	33809
query24	6627	2348	2333	2333
query25	474	463	390	390
query26	817	248	150	150
query27	2214	462	323	323
query28	5530	2473	2482	2473
query29	569	530	432	432
query30	211	184	145	145
query31	982	937	862	862
query32	91	68	63	63
query33	488	346	309	309
query34	782	836	521	521
query35	879	822	743	743
query36	1024	1042	944	944
query37	144	93	72	72
query38	4275	4497	4204	4204
query39	1528	1510	1436	1436
query40	211	119	101	101
query41	45	43	44	43
query42	117	109	104	104
query43	532	525	501	501
query44	1357	829	832	829
query45	182	177	172	172
query46	890	1073	652	652
query47	2032	2007	1943	1943
query48	404	425	319	319
query49	726	481	376	376
query50	649	696	397	397
query51	7285	7325	7212	7212
query52	106	97	93	93
query53	229	255	179	179
query54	494	503	400	400
query55	83	88	81	81
query56	258	250	236	236
query57	1260	1267	1188	1188
query58	239	219	227	219
query59	3236	3446	3235	3235
query60	293	267	253	253
query61	106	103	102	102
query62	842	826	774	774
query63	237	189	193	189
query64	3545	1064	652	652
query65	3325	3239	3233	3233
query66	812	404	312	312
query67	16380	15881	15624	15624
query68	10048	759	512	512
query69	485	284	252	252
query70	1268	1144	1135	1135
query71	425	282	239	239
query72	5924	3843	3777	3777
query73	658	746	358	358
query74	10269	9419	9190	9190
query75	4606	3161	2669	2669
query76	5493	1174	748	748
query77	1014	370	275	275
query78	10142	10209	9578	9578
query79	3278	876	590	590
query80	745	564	428	428
query81	489	261	221	221
query82	361	149	120	120
query83	198	160	152	152
query84	288	88	70	70
query85	750	361	293	293
query86	353	310	293	293
query87	4451	4416	4492	4416
query88	3183	2215	2270	2215
query89	430	337	288	288
query90	2123	182	183	182
query91	135	132	105	105
query92	66	52	50	50
query93	1997	867	519	519
query94	649	377	279	279
query95	334	259	249	249
query96	493	621	280	280
query97	2661	2826	2688	2688
query98	228	203	189	189
query99	1627	1570	1416	1416
Total cold run time: 299778 ms
Total hot run time: 197685 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.23 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4cb8fc8d501d7864772d6f9095f85b7a41c429b3, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.62	0.11	0.11
query5	0.43	0.44	0.41
query6	1.13	0.64	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.50	0.50
query10	0.55	0.56	0.55
query11	0.14	0.10	0.10
query12	0.12	0.11	0.11
query13	0.61	0.60	0.60
query14	2.70	2.82	2.85
query15	0.88	0.84	0.83
query16	0.38	0.40	0.35
query17	1.06	0.98	0.97
query18	0.22	0.21	0.21
query19	1.84	1.91	2.00
query20	0.02	0.01	0.01
query21	15.37	0.92	0.59
query22	0.77	0.75	0.63
query23	15.40	1.51	0.54
query24	3.22	1.63	0.87
query25	0.15	0.27	0.12
query26	0.28	0.14	0.13
query27	0.05	0.06	0.05
query28	13.69	1.50	1.04
query29	12.56	3.99	3.25
query30	0.24	0.09	0.06
query31	2.83	0.60	0.38
query32	3.23	0.54	0.47
query33	3.03	3.07	3.08
query34	16.68	5.09	4.49
query35	4.47	4.44	4.47
query36	0.62	0.48	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.74 s
Total hot run time: 31.23 s

@zzzxl1993
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32374 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f4059eb7cfdad028346cedcc967975bbe1893432, data reload: false

------ Round 1 ----------------------------------
q1	17576	6130	6024	6024
q2	2044	323	184	184
q3	10406	1254	727	727
q4	10244	880	448	448
q5	8278	2192	1968	1968
q6	207	182	143	143
q7	886	767	598	598
q8	9263	1327	1162	1162
q9	5273	4835	4930	4835
q10	6735	2312	1861	1861
q11	472	276	267	267
q12	335	352	222	222
q13	17764	3565	3014	3014
q14	239	241	211	211
q15	557	507	492	492
q16	618	603	592	592
q17	570	856	344	344
q18	6870	6470	6301	6301
q19	2356	979	564	564
q20	298	307	177	177
q21	2826	2116	1929	1929
q22	355	331	311	311
Total cold run time: 104172 ms
Total hot run time: 32374 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6331	6212	6238	6212
q2	237	331	235	235
q3	2255	2650	2281	2281
q4	1370	1798	1314	1314
q5	4377	4719	4815	4719
q6	188	175	138	138
q7	2072	1964	1784	1784
q8	2653	2771	2662	2662
q9	7330	7242	7348	7242
q10	3054	3385	2788	2788
q11	572	513	494	494
q12	668	780	642	642
q13	3381	3748	3155	3155
q14	282	302	294	294
q15	581	520	502	502
q16	662	708	667	667
q17	1220	1731	1263	1263
q18	7645	7446	7207	7207
q19	839	1212	1058	1058
q20	2005	2039	1876	1876
q21	5678	5254	4822	4822
q22	589	604	620	604
Total cold run time: 53989 ms
Total hot run time: 51959 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197258 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f4059eb7cfdad028346cedcc967975bbe1893432, data reload: false

query1	1278	974	895	895
query2	6422	2450	2303	2303
query3	10990	4801	4834	4801
query4	33378	23806	23940	23806
query5	4529	588	443	443
query6	280	206	211	206
query7	3985	504	303	303
query8	303	252	237	237
query9	9297	2711	2713	2711
query10	480	315	243	243
query11	18034	15322	15328	15322
query12	164	108	105	105
query13	1580	552	428	428
query14	10116	6788	6917	6788
query15	249	220	188	188
query16	7894	629	449	449
query17	1530	784	606	606
query18	2075	420	324	324
query19	245	182	162	162
query20	118	111	114	111
query21	204	124	107	107
query22	4683	4540	4467	4467
query23	34628	33812	33476	33476
query24	6387	2294	2334	2294
query25	484	458	390	390
query26	940	270	150	150
query27	2742	449	346	346
query28	5859	2487	2466	2466
query29	598	577	418	418
query30	220	191	155	155
query31	986	926	831	831
query32	73	55	57	55
query33	484	354	329	329
query34	779	860	523	523
query35	812	855	737	737
query36	1025	1052	970	970
query37	119	98	74	74
query38	4159	4162	4299	4162
query39	1531	1480	1489	1480
query40	216	114	102	102
query41	45	43	48	43
query42	117	103	105	103
query43	533	541	494	494
query44	1343	834	846	834
query45	190	179	162	162
query46	885	1076	668	668
query47	2009	1997	1948	1948
query48	385	415	333	333
query49	708	472	394	394
query50	674	677	417	417
query51	7274	7413	7330	7330
query52	103	104	91	91
query53	230	257	185	185
query54	486	505	421	421
query55	83	77	79	77
query56	268	263	238	238
query57	1253	1252	1136	1136
query58	241	221	232	221
query59	3259	3299	3215	3215
query60	270	271	264	264
query61	107	111	108	108
query62	870	809	745	745
query63	237	194	190	190
query64	3202	993	662	662
query65	3338	3272	3226	3226
query66	803	410	302	302
query67	16326	15803	15570	15570
query68	9434	754	516	516
query69	511	302	253	253
query70	1208	1169	1103	1103
query71	449	311	250	250
query72	5905	3845	3833	3833
query73	811	757	367	367
query74	10136	8999	9268	8999
query75	4651	3145	2678	2678
query76	5708	1182	776	776
query77	1051	382	273	273
query78	10077	10395	9646	9646
query79	2233	913	604	604
query80	641	519	466	466
query81	512	264	222	222
query82	501	149	125	125
query83	167	159	150	150
query84	243	103	71	71
query85	745	358	304	304
query86	341	315	293	293
query87	4384	4428	4510	4428
query88	3077	2244	2212	2212
query89	410	333	294	294
query90	1971	188	188	188
query91	134	137	106	106
query92	65	55	52	52
query93	930	876	522	522
query94	657	391	281	281
query95	322	271	250	250
query96	496	599	285	285
query97	2751	2835	2692	2692
query98	216	202	199	199
query99	1612	1539	1480	1480
Total cold run time: 297464 ms
Total hot run time: 197258 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.89% (10120/26021)
Line Coverage: 29.89% (85556/286196)
Region Coverage: 29.03% (43722/150626)
Branch Coverage: 25.56% (22304/87268)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f4059eb7cfdad028346cedcc967975bbe1893432_f4059eb7cfdad028346cedcc967975bbe1893432/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 32.23 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f4059eb7cfdad028346cedcc967975bbe1893432, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.08	0.07
query4	1.61	0.11	0.10
query5	0.42	0.43	0.41
query6	1.14	0.65	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.59	0.48	0.53
query10	0.54	0.58	0.56
query11	0.14	0.10	0.11
query12	0.14	0.12	0.11
query13	0.62	0.60	0.60
query14	2.72	2.77	2.81
query15	0.90	0.82	0.82
query16	0.41	0.38	0.38
query17	1.02	1.06	1.07
query18	0.22	0.22	0.21
query19	1.87	1.78	1.98
query20	0.01	0.01	0.01
query21	15.35	0.92	0.58
query22	0.75	0.83	0.75
query23	15.18	1.46	0.59
query24	3.04	1.72	2.18
query25	0.21	0.16	0.09
query26	0.18	0.14	0.13
query27	0.05	0.08	0.05
query28	14.69	1.45	1.05
query29	12.55	3.97	3.30
query30	0.25	0.10	0.06
query31	2.83	0.60	0.40
query32	3.23	0.56	0.46
query33	3.13	3.24	3.06
query34	16.90	5.15	4.52
query35	4.52	4.47	4.46
query36	0.66	0.52	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.03
query40	0.17	0.14	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.74 s
Total hot run time: 32.23 s

@zzzxl1993 zzzxl1993 changed the title [opt](inverted index) inverted Index File Cache Queue Optimization [fix](inverted index) inverted Index File Cache Queue Optimization Jan 2, 2025
airborne12
airborne12 approved these changes Jan 3, 2025
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 3, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 3, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 3, 2025

PR approved by anyone and no changes requested.

qidaye
qidaye approved these changes Jan 3, 2025
View reviewed changes
Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qidaye qidaye merged commit 0c81764 into apache:master Jan 3, 2025
25 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@qidaye qidaye qidaye approved these changes

@airborne12 airborne12 airborne12 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.x dev/3.0.x-conflict reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@zzzxl1993 @hello-stephen @doris-robot @airborne12 @qidaye